Segmenting multiple overlapping objects via a hybrid active contour model incorporating shape priors: applications to digital pathology

Active contours and active shape models (ASM) have been widely employed in image segmentation. A major limitation of active contours, however, is in their (a) inability to resolve boundaries of intersecting objects and to (b) handle occlusion. Multiple overlapping objects are typically segmented out as a single object. On the other hand, ASMs are limited by point correspondence issues since object landmarks need to be identified across multiple objects for initial object alignment. ASMs are also are constrained in that they can usually only segment a single object in an image. In this paper, we present a novel synergistic boundary and region-based active contour model that incorporates shape priors in a level set formulation. We demonstrate an application of these synergistic active contour models using multiple level sets to segment nuclear and glandular structures on digitized histopathology images of breast and prostate biopsy specimens. Unlike previous related approaches, our model is able to resolve object overlap and separate occluded boundaries of multiple objects simultaneously. The energy functional of the active contour is comprised of three terms. The first term comprises the prior shape term, modeled on the object of interest, thereby constraining the deformation achievable by the active contour. The second term, a boundary based term detects object boundaries from image gradients. The third term drives the shape prior and the contour towards the object boundary based on region statistics. The results of qualitative and quantitative evaluation on 100 prostate and 14 breast cancer histology images for the task of detecting and segmenting nuclei, lymphocytes, and glands reveals that the model easily outperforms two state of the art segmentation schemes (Geodesic Active Contour (GAC) and Roussons shape based model) and resolves up to 92% of overlapping/occluded lymphocytes and nuclei on prostate and breast cancer histology images.